Head mounted display and information processing method

ABSTRACT

Methods and apparatus providing a head-mounted display to be worn by the user and presenting a video to the user through the head-mounted display; measuring outside world information via an outside world measurement section of the head mounted display; detecting, via a notification information detection section, whether or not the information measured by the outside world measurement section contains any notification information to be notified to the user, including that the notification information detection section detects at least one reference position, which is set by the user as notification information; and notifying the user, via a notification section, when the notification information detection section detects notification information by way of displaying a video of the at least one reference position to the user in the head-mounted display.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.:15/501,372, accorded a filing date of Feb. 2, 2017 (allowed), which is aNational Phase Application of PCT/JP2015/080383, filed Oct. 28, 2015,which is an international application claiming priority to JP2014-224154, filed Nov. 4, 2014, the entire disclosures of which arehereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to a head-mounted display and aninformation processing method carried out by the head-mounted display.

BACKGROUND ART

Recent years have seen advancement in technological developments forpresenting stereoscopic videos, contributing to widespread use ofhead-mounted displays that permit presentation of stereoscopic videoswith depth. Capable of presenting videos having depth to users,head-mounted displays are now used as presentation devices ofthree-dimensional (3D) content such as 3D movies and 3D games.

SUMMARY

[Technical Problem]

A head-mounted display is worn in such a manner as to cover botheyeballs of the user, an observer. Some head-mounted displays arenon-see-through ones. Each of these head-mounted displays blocks out anoutside world from a user's vision and presents a video when worn by theuser. In general, when wearing such a non-see-through head-mounteddisplay, the user has difficulty in viewing a real-world scene in theoutside world.

For example, if a non-see-through head-mounted display is used where theuser is allowed to move or play to a limited extent such as at home, itis convenient to present, to the user, the extent to which the userwearing the head-mounted display is allowed to move.

The present invention has been devised in light of the foregoing, and itis an object of the present invention to provide a technology fornotifying an outside world condition to a user wearing a non-see-throughhead-mounted display.

[Solution to Problem]

In order to solve the above problem, a head-mounted display according toa mode of the present invention blocks out an outside world from auser's vision and presents a video when worn by the user. Thehead-mounted display includes an outside world measurement section, anotification information detection section, and a notification section.The outside world measurement section measures outside worldinformation. The notification information detection section detectswhether or not the information measured by the outside world measurementsection contains any notification information to be notified to theuser. The notification section notifies the user when the notificationinformation detection section detects notification information.

Another mode of the present invention is an information processingmethod carried out by a processor of a head-mounted display for blockingout an outside world from a user's vision when worn by the user topresent a video. This method includes a step of measuring outside worldinformation. This method includes another step of detecting whether ornot the measured information contains any notification information to benotified to the user. This method includes still another step ofnotifying the user when notification information is detected.

Still another mode of the present invention is a program for causing acomputer to perform the steps of the above method.

This program may be supplied as part of firmware built into an apparatusto basically control hardware resources such as video, audio, gameapparatus, and head-mounted display. This firmware is stored in asemiconductor memory such as read only memory (ROM) or flash memory inthe apparatus. In order to supply this firmware or to update part of thefirmware, a computer-readable recording medium recording this programmay be supplied or transmitted via a communication line.

It should be noted that arbitrary combinations of the above componentsand conversions of expressions of the present invention between method,device, system, computer program, data structure, recording medium andso on are also effective as modes of the present invention.

[Advantageous Effect of Invention]

The present invention provides a technology for notifying an outsideworld condition to a user wearing a non-see-through head-mounteddisplay.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically illustrating an overview of anappearance configuration of a head-mounted display according to anembodiment.

FIG. 2 is a diagram schematically illustrating a functionalconfiguration of the head-mounted display according to the embodiment.

FIG. 3 is a diagram schematically illustrating a scene in which thehead-mounted display according to the embodiment is used.

FIG. 4 is a flowchart describing an information processing flow handledby the head-mounted display according to the embodiment.

DESCRIPTION OF EMBODIMENT

FIG. 1 is a diagram schematically illustrating an overview of anappearance configuration of a head-mounted display 100 according to anembodiment, and shows that a user 1 is wearing the head-mounted display100. It is assumed that the head-mounted display 100 according to theembodiment is a non-see-through head-mounted display. Therefore, whenthe user 1 wears the head-mounted display 100, the head-mounted display100 blocks out an outside world from a vision of the user 1 asillustrated in FIG. 1. This allows the user 1 to focus on content videospresented by the head-mounted display 100, thus enhancing so-called“immersion.”

The head-mounted display 100 includes a housing 110 and a fitting 120.The fitting 120 is a member used to fasten the housing 110 to a head ofthe user 1 and can be implemented, for example, by a tightening orelastic belt. The housing 110 accommodates a central processing unit(CPU) which is not shown, a movie presentation section 130 a, and anaudio presentation section 130 b. The movie presentation section 130 aincludes, for example, a liquid crystal display (LCD) or organicelectro-luminescence (EL). The audio presentation section 130 bincludes, for example, a speaker and an earphone. It should be notedthat, in the present specification, a “video” includes a “movie,” timelysuccessive images, and “audio” that is played in synchronism with orindependently of the movie. The head-mounted display 100 furtherincludes an outside world measurement section 140 that measuresinformation on an outside world of the housing 110. The outside worldmeasurement section 140 will be described in detail later.

FIG. 2 is a diagram schematically illustrating a functionalconfiguration of the head-mounted display 100 according to theembodiment. The head-mounted display 100 includes an informationpresentation section 130, the outside world measurement section 140, anotification information detection section 150, a notification section160, a position measurement section 170, a control section 180, and avideo playing section 190.

FIG. 2 illustrates a functional configuration for enabling thehead-mounted display 100 according to the embodiment to notify anoutside world condition to the user 1. Other part of the configurationis not shown. In FIG. 2, each of the elements described as a functionalblock for performing various processing tasks can be formed with a CPU,a main memory, and other large scale integrations (LSIs) in terms ofhardware. In terms of software, on the other hand, each of the elementsis implemented by a program loaded into the main memory. Therefore, itis understood by those skilled in the art that these functional blockscan be implemented in various ways by hardware alone, software alone, ora combination thereof and are not limited to any one of them.

The video playing section 190 plays the content video presented to theuser 1. The video playing section 190 plays the content acquired via anunshown memory card slot or universal serial bus (USB) or by wirelessmeans such as wireless fidelity (Wi-Fi) (registered trademark). Theinformation presentation section 130 includes the movie presentationsection 130 a and the audio presentation section 130 b described above,presenting the video played by the video playing section 190 to the user1. The information presentation section 130 causes the moviepresentation section 130 a to present movies, still images, and otherinformation and the audio presentation section 130 b to present audioinformation. The information presentation section 130 functions as auser interface for presenting information generated by the head-mounteddisplay 100 to the user 1.

The outside world measurement section 140 measures information on anoutside world of the housing 110. The outside world measurement section140 is provided at a position on the outside of the housing 110 of thehead-mounted display 100 that matches the position in which the moviepresentation section 130 a is accommodated. Therefore, when thehead-mounted display 100 is worn by the user 1, the outside worldmeasurement section 140 can measure outside world information in thedirection of line of sight of the user 1.

The outside world measurement section 140 can be implemented, forexample, by a camera designed to measure visible radiation or aninfrared camera designed to measure infrared radiation. It should benoted that the outside world measurement section 140 may be a stereocamera with at least two cameras. This makes it possible to acquiredepth information of a subject by analyzing information measured by theoutside world measurement section 140.

The notification information detection section 150 detects whether ornot any notification information to be notified to the user 1 iscontained in the information measured by the outside world measurementsection 140. “Information to be notified” refers to information used toidentify the extent to which the user 1 is allowed to move. Morespecifically, if the outside world measurement section 140 is a stereocamera, the notification information detection section 150 detects stillobjects that stand still of all subjects measured by the outside worldmeasurement section 140, a stereo camera, as notification information.This can be implemented by a known 3D image analysis technology.

The notification section 160 notifies the user 1 when notificationinformation is detected by the notification information detectionsection 150. For example, the notification section 160 may show, on theinformation presentation section 130, notification information to theeffect that a still object exists in the outside world in the form of amessage. Alternatively, a window showing a video of the still object maybe displayed on the information presentation section 130. Thenotification section 160 may cause the information presentation section130 to play notification information to the effect that a still objectexists in the outside world in the form of audio information.Alternatively, the notification section 160 may cause the informationpresentation section 130 to play a warning tone indicating that a stillobject exists in the outside world. If the head-mounted display 100 hasa vibrator (not shown), the user 1 may be notified by vibration.

The position measurement section 170 detects where the head-mounteddisplay 100 exists. More specifically, the position measurement section170 measures the position of the head-mounted display 100 in the spacewhere the head-mounted display 100 exists. The position measurementsection 170 includes, for example, an acceleration sensor and an angularspeed sensor, thus detecting a relative position of the head-mounteddisplay 100 with respect to a reference point determined in advance bythe user 1.

The control section 180 comprehensively controls each of the differentsections of the head-mounted display 100 described above. The controlsection 180 is implemented as a result of execution of the operatingsystem of the head-mounted display 100 by the CPU (not shown) of thehead-mounted display 100.

FIG. 3 is a diagram schematically illustrating a scene in which thehead-mounted display 100 according to the embodiment is used. Theexample in FIG. 3 shows that the user 1 is wearing the head-mounteddisplay 100 in a room where a desk 200 exists.

We consider, for example, a case in which the user 1 wears thehead-mounted display 100 and plays a 3D game. In this case, the user 1advances through the game by manipulating a controller of a gamingdevice (not shown). The head-mounted display 100 presents a video thatchanges in accordance with the progress of the game to the user 1.

As described above, the head-mounted display 100 according to theembodiment is a non-see-through head-mounted display that presents 3Dvideos full of immersion to the user 1. Therefore, what the user 1perceives from the videos presented by the head-mounted display 100 is arealistic experience. In some cases, the user 1 may move his or her heador body reflexively. When the user uses the head-mounted display 100 ina relatively small space such as a room at home, the head-mounteddisplay 100 can be made easier to use if the extent to which the user 1is allowed to move, i.e., the area within which the user 1 can enjoy thegame, can be presented. This can enhance the usability of thehead-mounted display 100.

For this reason, when the notification information detection section 150detects a still object, the notification to the user 1 by thenotification section 160 may be triggered by the fact that the distancebetween the head-mounted display 100 and the still object becomesshorter than a given reference distance.

Here, the term “given reference distance” refers to a “notificationreference distance” that serves as a reference for deciding whether ornot the notification section 160 should notify notification informationto the user 1. Although may be determined experimentally inconsideration of the anticipated manner in which the head-mounteddisplay 100 is used and so on, the notification reference distance is,for example, one meter. The notification section 160 notifies the user 1accordingly if the distance between the still object and thehead-mounted display 100 becomes shorter than one meter. This allows theuser 1 to recognize that he or she has come close to the boundary of themovable area.

If the movement of the head-mounted display 100 is negligible, forexample, as when the user 1 uses the head-mounted display 100 seated,the still object is shot as an immobile subject in the video shot by theoutside world measurement section 140. In such a case, the notificationinformation detection section 150 need only identify the still object byanalyzing the video measured by the outside world measurement section140. Further, if the outside world measurement section 140 is a stereocamera, the notification information detection section 150 can estimatethe distance to the still object by analyzing the video measured by theoutside world measurement section 140.

In contrast, if the user 1 uses the head-mounted display 100, forexample, while standing and moving, the movement of the head-mounteddisplay 100 is not negligible. In this case, analyzing the videomeasured by the outside world measurement section 140 alone may lead tolow accuracy in identifying the still object. The reason for this isthat the position of the still object, despite standing still object,changes in the video measured by the outside world measurement section140 because of the movement of the head-mounted display 100.

Therefore, the notification section 160 may acquire the distance betweenthe head-mounted display 100 and the still object based on the positionof the head-mounted display 100 detected by the position measurementsection 170. In this case, the user 1 determines a reference point forposition measurement in advance before using the head-mounted display100. In the example shown in FIG. 3, the user 1 determines a point P, apoint in the desk 200, as the reference point P.

As illustrated in FIG. 3, the user 1 establishes a Cartesian coordinatesystem 300 with its origin set at an arbitrary point in the space wherethe user 1 and the desk 200 exist. The position of the reference point Pcan be set uniquely as position coordinates of the Cartesian coordinatesystem 300. In FIG. 3, the coordinates of the reference point P are setat P(X₁, Y₁, Z₁). The user 1 moves the head-mounted display 100 to theposition of the reference point P first and then instructs thehead-mounted display 100 to start measurement. This can be accomplished,for example, by pressing a measurement start button (not shown) attachedto the housing 110.

The position measurement section 170 includes an acceleration sensor andan angular speed sensor. The position measurement section 170 integratesoutput values of the sensors after being instructed to start measurementby the user 1, thus allowing for calculation of the relative position ofthe head-mounted display 100 with respect to the reference point P.Further, the position measurement section 170 can acquire the positioncoordinates of the head-mounted display 100 in the Cartesian coordinatesystem 300 by merging the relative position of the head-mounted display100 with the coordinates P(X₁, Y₁, Z₁) of the reference point P. Inparticular, if the user 1 specifies the reference point P as a point onthe still object, the notification section 160 can calculate thedistance between the still object and the head-mounted display 100without using the coordinates P(X₁, Y₁, Z₁) of the reference point P.

By acquiring the position coordinates of the head-mounted display 100,i.e., the movement of the head-mounted display 100, the notificationinformation detection section 150 can identify the reason why thesubject measured by the outside world measurement section 140 moved inthe video. This provides improved accuracy in extraction of stillobjects by the notification information detection section 150. Further,the distance between the still object and the head-mounted display 100can be acquired with higher accuracy than by acquiring depth informationof the subject through analysis of information shot by the outside worldmeasurement section 140, a stereo camera.

FIG. 4 is a flowchart describing an information processing flow handledby the head-mounted display 100 according to the embodiment. Theprocesses in this flowchart begin, for example, when the head-mounteddisplay 100 is powered on.

The outside world measurement section 140 measures outside worldinformation in the direction of line of sight of the user 1 when thehead-mounted display 100 is worn by the user 1 (S2). While a stillobject remains unfound by the notification information detection section150 through analysis of the video measured by the outside worldmeasurement section 140 (N in S4), the outside world measurement section140 continues with measurement of the outside world.

When a still object is found in the video measured by the outside worldmeasurement section 140 (Y in S4), the notification informationdetection section 150 sets the detected still object as notificationinformation (S6). The notification section 160 acquires a distance Lfrom the head-mounted display 100 to the still object (S8) based on theposition of the head-mounted display 100 detected by the positionmeasurement section 170.

When the distance L from the head-mounted display 100 to the stillobject is equal to or longer than a given reference distance L_(T) (N inS10), the notification section 160 returns to step S8 to continue withacquisition of the distance L. If the distance L from the head-mounteddisplay 100 to the still object is shorter than the given referencedistance L_(T) (Y in S10), the notification section 160 notifies theuser via the information presentation section 130 that a still object isclose (S12).

While the user 1 continues to use the head-mounted display 100 (N inS14), the head-mounted display 100 returns to step S2 to continue withthe series of processes described above. When the user 1 stops using thehead-mounted display 100 (Y in S14), the processes in this flowchart areterminated.

As described above, the head-mounted display 100 according to theembodiment can provide a technology for notifying an outside worldcondition to the user 1 wearing the non-see-through head-mounteddisplay.

In particular, if the head-mounted display 100 worn by the user 1 comesclose to a still object in the outside world, information to that effectis presented on the information presentation section 130 of thehead-mounted display 100. This allows the user 1 to find out that he orshe has come close to the boundary of the area within which the user 1is allowed to freely move even with the non-see-through head-mounteddisplay 100 on.

Thus, the present invention has been described based on the embodiment.It is to be understood by those skilled in the art that the embodimentis illustrative, that combinations of components and processes thereofcan be modified in various ways, and that these modification examplesalso fall within the scope of the present invention.

In the above description, a case was described in which the notificationinformation detection section 150 detected a still object in outsideworld information measured by the outside world measurement section 140as notification information. Instead or in addition thereto, thenotification information detection section 150 may detect, asnotification information, an object moving at a given speed or more inoutside world information measured by the outside world measurementsection 140.

For example, if the user 1 uses the head-mounted display 100 accordingto the embodiment in a room with an inwardly opening door, the areawithin which the user 1 is allowed to freely move changes when the dooris opened by somebody other than the user 1. That is, if the inwardlyopening door opens, this means that the door enters into the room, thusresulting in a smaller area within which the user 1 is allowed to move.For this reason, the unshown notification information detection section150 detects a moving object in the outside world information measured bythe outside world measurement section 140, notifying the result thereofto the user 1.

This allows the head-mounted display 100 according to a modificationexample to notify any change in the outside world condition to the user1 wearing the non-see-through head-mounted display. Even in the event ofa change in the area within which the user 1 is allowed to move, thehead-mounted display 100 can adaptively notify the boundary thereof tothe user 1.

REFERENCE SIGNS LIST

100 Head-mounted display, 110 Housing, 120 Fitting, 130 Informationpresentation section, 130 a Movie presentation section, 130 b Audiopresentation section, 140 Outside world measurement section, 150Notification information detection section, 160 Notification section,170 Position measurement section, 180 Control section, 190 Video playingsection, 200 Desk, 300 Cartesian coordinate system.

INDUSTRIAL APPLICABILITY

The present invention is applicable to information processingtechnologies for head-mounted displays.

The invention claimed is:
 1. A head-mounted display for blocking out anoutside world from a user's vision when worn by the user to present avideo, the head-mounted display comprising: an outside world measurementsection adapted to measure outside world information; a notificationinformation detection section adapted to detect whether or not theinformation measured by the outside world measurement section containsany notification information to be notified to the user, including thatthe notification information detection section detects at least onereference position, which is set by the user as notificationinformation; and a notification section adapted to notify the user whenthe notification information detection section detects notificationinformation by way of displaying a video of the at least one referenceposition to the user in the head-mounted display.
 2. The head mounteddisplay of claim 1, wherein the at least one reference position isdetected by the system using an object moved by the user.
 3. The headmounted display of claim 1, wherein the at least one reference positionis detected by the system using an object worn by the user.
 4. The headmounted display of claim 3, wherein the object worn by the user is thehead mounted display.
 5. The head mounted display of claim 1, whereinthe at least one reference position is detected by the system using anobject moved by the user, which is the head mounted display.
 6. The headmounted display of claim 1, wherein the at least one reference positionis detected by the system using a user-controlled object, whereby theuser presses a button on the user-controlled object.
 7. The head mounteddisplay of claim 1, wherein the user sets the at least one referenceposition in advance of using the head-mounted display.
 8. The headmounted display of claim 1, wherein the notification is triggered when adistance between the head-mounted display and the at least one referenceposition becomes shorter than a reference distance.
 9. The head mounteddisplay of claim 1, wherein the outside world measurement section isattached to the head-mounted display.
 10. A method of blocking out anoutside world from a user's vision, comprising: providing a head-mounteddisplay to be worn by the user and presenting a video to the userthrough the head-mounted display; measuring outside world informationvia an outside world measurement section of the head mounted display;detecting, via a notification information detection section, whether ornot the information measured by the outside world measurement sectioncontains any notification information to be notified to the user,including that the notification information detection section detects atleast one reference position, which is set by the user as notificationinformation; and notifying the user, via a notification section, whenthe notification information detection section detects notificationinformation by way of displaying a video of the at least one referenceposition to the user in the head-mounted display.
 11. The method ofclaim 10, wherein the at least one reference position is detected by thesystem using an object moved by the user.
 12. The method of claim 10,wherein the object moved by the user is worn by the user.
 13. The methodof claim 12, wherein the object worn by the user is the head mounteddisplay.
 14. The method of claim 10, wherein the object moved by theuser is the head mounted display.
 15. The method of claim 10, whereinthe at least one reference position is detected by the system using auser-controlled object, whereby the user presses a button on theuser-controlled object.
 16. The method of claim 10, wherein the usersets the at least one reference position in advance of using thehead-mounted display.
 17. The method of claim 10, wherein thenotification is triggered when a distance between the head-mounteddisplay and the at least one reference position becomes shorter than areference distance.
 18. The method of claim 10, wherein the outsideworld measurement section is attached to the head-mounted display.
 19. Anon-transitory, computer readable storage medium containing a computerprogram, which when executed by a computer, operates to carry out amethod of blocking out an outside world from a user's vision, the methodcomprising: providing a head-mounted display to be worn by the user andpresenting a video to the user through the head-mounted display;measuring outside world information via an outside world measurementsection of the head mounted display; detecting, via a notificationinformation detection section, whether or not the information measuredby the outside world measurement section contains any notificationinformation to be notified to the user, including that the notificationinformation detection section detects at least one reference position,which is set by the user as notification information; and notifying theuser, via a notification section, when the notification informationdetection section detects notification information by way of displayinga video of the at least one reference position to the user in thehead-mounted display.